In recent years, a demand for a monitoring camera for security, site management, or the like has increased, and with the development of semiconductors and network technology, functions of the monitoring camera have been diversified.
From among those, image correction represented by shake correction, gradation correction, or removal of disturbances, for example, fog, rain, snowstorm, yellow dust, smog, or the like, plays an important role in a monitoring field which is easily affected by the atmosphere and illumination change, and until now, has been provided in various products.
Such an image correction is aimed to improve image visibility and increase monitoring efficiency. For example, technology of the shake correction (stabilizer) detects extra movement of an image caused by strong wind or vibration of a camera fixing stand and performs correction in such a way to cancel the extra movement, thereby providing an image having a high visibility.
As one of natural phenomena that considerably decrease the visibility of a monitored image, heat haze can be mentioned. The heat haze is a phenomenon in which refraction/traveling of light is obstructed, occurring due to the temperature difference of the air, or the like. In a place in which heat haze occurs, a subject is deformed as lint balls stand and is viewed as irregularly shaking, and resolution is also reduced. Therefore, the quality of the image is considerably reduced.
In practice, attempts to correct image distortion caused by heat haze or to increase resolution of an image with heat haze have been made.
Relevant technology is to smooth all pixels of a screen by using a plurality of temporal neighboring images (hereinafter, referred to as “time-series smoothing”) and relieve deformation of the subject due to heat haze.
That is, time-series smoothing is technology which obtains an image close to an original shape by performing pixels using a plurality of images since it is considered that a displacement amount in a case where the subject is deformed due to the influence of heat haze, statistically has a property according to Gaussian distribution.
For example, referring to Non-patent Literature 1, there is disclosed technology of correcting deformation of a subject due to heat haze by performing position matching on an “input image” that is a target for heat haze correction and an “average image” which is time-series smoothed, and performing correction such that the input image is matched on the average image.
An existing method of correcting heat haze by performing time-series smoothing will be described with reference to FIG. 11.
Input images 1200-1 to 1200-n represent an image group of a plurality of frames which are captured by a monitoring camera or the like and are temporally adjacent to each other. FIG. 11 illustrates the input images 1200-1 to 1200-4 as examples. When such an image includes distortion due to heat haze, the shape of a subject is deformed around a place in which heat haze occurs, thereby leading to reduction in video visibility.
A time-series smoothed image 1220 is an example of an image obtained by smoothing all pixels of a screen by performing smoothing on the input images 1200-1 to 1200-4 in an existing time-series of an image. As in the time-series smoothed image 1220, it is possible to restore an image close to an original shape of a body in a background region in which a body, such as a building, is stationary.
However, when time-series smoothing is performed in a case where a moving body is included in the input image, an image in which a moving body region is unnaturally blurred is generated, and a corrected image is considerably deteriorated. That is, in the example of FIG. 11, there is a problem that, in a region 1250 through which the moving body has passed, the moving body and the region are unintentionally superimposed on each other and a desired result is not obtained.
That is, the time-series smoothing method can be used only when the moving body is not present in the screen.
Also, since resolution is also reduced by the above-described smoothing processing, in addition to missing of resolution due to the influence of heat haze, an attempt to increase resolution of the corrected image has been made.
For example, referring to Non-patent Literature 2, there is disclosed a technology of restoring missed resolution by performing a super-resolution process using a plurality of temporal neighboring images on an image obtained by performing time-series smoothing processing on all pixels of the screen.
Also, referring to Patent Literature 1, there is disclosed an image processing apparatus (hereinafter referred to as related art 3) including a correcting device that performs an operation of applying a transfer function, which is defined by a single coefficient and the image data of a preceding frame, to image data in order to correct image fluctuation in the input image data, a delay device that delays the image data corrected by the correcting device by one frame and feeds the image data back to the correcting device as the image data of the preceding frame, an extracting device that extracts pixels in which movement is present on the image, based on input image data and the image data of the preceding frame, an identifying device that identifies a pixel of a moving body from pixels having fluctuation among the pixels in which the movement is present, and a control device that controls the coefficient according to a result of the identification by the identifying device.
The related art 3 also can reduce influence affecting a moving body and correct image fluctuation.